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(57) Abstract 

An interactive eyewear selection system comprising a video camera (18). and digitizing system (24) for acquiring a digital image 
of a person's face, including the temple areas of the face; an eyeglass frame database (16) for storing digital eyeglass frame images; an 
automated interface for selecting an eyeglass frame image from the frames database, the image contammg frame temple portions; a digital 
processing system (38) for automatically superimposing the selected frame image on the image of the person's face, in a composite image, 
such that the frame appears to be worn naturally on the person's face, and the frame temple portions are shown at the temple areas of the 
face and a display device (26) for displaying the digital image of the person's face, and for displaying the composite image. 



FOR THE PURPOSES OF INFORMATION ONLY 
Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Annenia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BE 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 
Burkina Paso 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turkmenistan 


BF 


GR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 
UA 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


Ukraine 


BR 
BY 


Brazil 


IL 


Israel 


MR 


Mauritania 


VG 


Uganda 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


' Canada 


IT 


Italy 


MX 


Mexico 


uz 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Vict Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzsian 


NO 


Norway 


ZW 


Zimbabwe 


CI 


Catc d'l voire 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


FT 


Portugal 






cu 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






cz 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






I>E 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


t>cnmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SC 


Singapore 







SDOCID <WO 0016683A1 t > 



wo 00/16683 



PCT/US99/21918 



INTERACTIVE EYEWEAR SELECTION SYSTEM 

BACKGROUND OF THE INVENTION 

1. T/=.r^hnica1 Field 

The present invention relates to the optometry and optician 
fields, and more particularly to digital image processing and 
displaying apparatus for use by optometrists, opticians, 
eyeglass vendors, and related parties. 

2. Ptar-Vcr-round Art 

When a person requires a new pair of eyeglasses, he or she 
usually visits an optometrist for an eye examination. Normally, 
an optician is also on hand to assist the person in selecting 
and fitting the new eyeglasses. After the eye exam, the person 
must select a frame for his or her eyeglasses. As many of us 
know, this process involves a review of many different frames 
made physically available to the person by the optician. 

This process can be tedious, because: (1) the person may 
have to wait before being helped by an optician; (2) the person 
must choose from a large number of frame styles and 
manufacturers, usually requiring the assistance of the optician; 
(3) the person must physically try-on each of the frames 
selected for consideration; and (4) the person must preview each 
tried-on frame in a mirror without his or her prescription 
lenses, which can be difficult for some and inconvenient for 
most . 

Also, when viewing frames on the face, the person does not 
see the optical effects of the prescription lenses to be placed 
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in the frames. Thus, the person must select a frame without 
assessing this important aesthetic factor. 

The above -described process requires the optician to 
display a large physical inventory of frames. This requirement 

5 is expensive, and inevitably limits the number and variety of 
frames that an optician can make available to the customer. 

Once a particular frame is selected, the optician must then 
obtain certain measurements needed to produce the eyeglasses 
(i.e., the position of the customer's eyes with respect to the 

10 frame) . • These measurements include the distance between the 

pupils, the ocular centers, the monocular pupillary, distances, 
and the segment heights. Such measurements are critical for the 
proper manufacture of the eyeglasses for the selected frames. 
If such measurements are not properly taken, the resulting 

15 eyeglasses become useless to the customer, and time and money 

are wasted. Such measurements are usually taken manually with a 
ruler or with the aid of various measurement devices. All such 
methods require skill and great care to achieve acceptable 
accuracy. In practice, such accuracy is not always attained. 

2 0 Different approaches have been proposed to overcome the 

various drawbacks mentioned above with respect to frame 
selection and eye/frame measurement. One particularly 
interesting approach is to simulate the frames (or eyewear) in a 
computer and digitally superimpose such eyewear on a digital 

2 5 image of the customer. Systems to carry out such an approach 

are proposed in the following patents: U.S. Patent 5,280,570 to 
Jordan; U.S. Patent 4,8 52,184 to Tamura et al . ; U.S. Patent 
4,845,641 to Ninomiya et al . ; and U.S. Patent 4,730,260: to Mori 
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et al . These computer-based systems have not gained widespread 
acceptance in the eyewear industry and by general public. 

One reason for this lack of acceptance is that such systems 
may not be presenting a realistic composite image of the 
customer wearing a selected frame. For instance, the systems 
proposed in U.S. Patents 5,280,570 to Jordan, 4,852,184 to 
Tamura et al . , 4,845,641 to Ninomiya et al . , and 4,73 0,260 to 
Mori et al . , do not display the temple portions of the frames on 
the face. The absence of these temple portions is immediately 
apparent to the customer, and adversely affects the credibility 
of the system as perceived by the customer. As a result, the 
customer is not likely to rely on such a system for selecting 
frames . 

U.S. Patent 4,53 9,585 to Spackova et al . proposes merging a 
frame having a temple portion with a face image, using 
separately displayed images of the frame and face and combining 
them with a half mirror. A digital processor embodiment is also 
proposed, however, it is not explained how the frame and face 
images are to be merged in the digital processor. 

Further, as to the realism issue, the systems proposed in 
U.S. Patents 5,280,570 to Jordan, 4,852,184 to Tamura et al . , 
4,845,641 to Ninomiya et al . , and 4,539,585 to Spackova et al., 
do not address the problem of superimposing transparent and 
semi-transparent frames and frame parts (such as the nosepad) on 
a face image. Transparent and semi-transparent frames (or parts 
thereof) pickup background color when the frame image is 
created. This background color is very noticeable in the frame 
(or frame part) when viewing the composite image, and thus 
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conveys a fake and not very convincing appearance. U.S. patent 
4,730,260 to Mori et al . recognizes this problem, however, it 
fails to describe any real solution. 

Another possible reason why the above-mentioned patented 
5 systems have not gained widespread acceptance is that such 

systems do not provide automatic measurement of the eye/frame 
parameters necessary to manufacture the lenses for the selected 
frame. U.S. Patent 5,617,155 to Ducarouge et al . proposes a 
system to automatically obtain the eye/frame measurements by way 

10 of pupil and frame detection. However, in Ducarouge et al . , the 
customer must wear a selected frame before his or her image is 
acquired by the system. Thus, the customer must choose his or 
her frame in the conventional manner before the system can be 
used, and the optician must carry a full complement of frames as 

15 is conventionally done. Similarly, in U.S. Patent 5,592,248 to 
Norton et al . , the customer must wear a selected frame before 
his or her image is acquired and measurements are made. 

In U.S. Patent 5,280,570 to Jordan it is mentioned that the 
shape of the customer's face is an important factor in selecting 

20 frames. However, in Jordan, the customer or optician must first 
determine the shape of the customer's face and then manually 

enter this parameter into the system. Such a process may be 
inconvenient and may yield an incorrect face shape 
determination. For example, it may be difficult for an 
2 5 inexperienced optician to distinguish between oblong and oval 
faces. 

U.S. Patent 5., 576,778 to Fujie et al . proposes a system 
which acquires the face image of the customer and, through 
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digital image detection, automatically determines the face shape 
and other facial features of the customer. The system then 
assists the customer in selecting frames that are best suited 
for his or her facial features. However, the Fujie et al . 
5 system does not simulate and superimpose eyeglass frames on the 
face image. Thus, the optician must carry a full complement of 
frames as is conventionally done. 

OBJECTS AND SUMMARY OF THE INVENTION 

It is therefore an object of the present invention to 
10 provide apparatus and methods that overcome the problems, 
drawbacks and limitations of the prior art. 

It is another object of the present invention to provide 
apparatus and methods that aid the optician ' s customer in 
selecting a suitable frame for his or her eyeglasses. 
15 It is a further object of the present invention to provide 

apparatus and methods that substantially eliminate the tedium 
normally associated with selecting eyeglass frames. 

It is yet another object of the present invention to 
substantially reduce the physical inventory of frames that an 
20 optician must carry. 

It is yet a further object of the present invention to 
provide apparatus and methods that simulate eyeglass frames and 
digitally superimpose such frames on a digital image of the 
customer, to create a composite digital image. 
25 It is still another object of the present invention to 

provide a realistic composite digital image of the eyeglass 
frames being worn by the customer. 
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It' is still a further object of the present invention to 
provide apparatus and methods that simulate the complete 
eyeglass frame, including the temple portions. 

It is yet still another object of the present invention to 
simulate transparent and semi-transparent eyeglass frames and 
frame parts. 

It is yet still a further object of the present invention 
to conveniently provide a wide variety of eyeglass frames to the 
customer for consideration. 

It is yet still a further object of the present invention 
to provide apparatus and methods that automatically detect the 
irises and pupil centers in a digital face image of a person. 

It is yet still a further object of the present invention 
to provide apparatus and methods that automatically detect the 
shape of a person » s face from a digital image of the person's 
face , 

It is yet still a further object of the present invention 
to provide apparatus and methods that automatically measure the 
eye/frame parameters needed to produce eyeglasses. 

It is yet still a further object of the present invention 
to provide apparatus and methods that simulate contact lenses on 
the eyes of a person, in a digital image of the person. 

It is yet still a further object of the present invention 
to provide an interactive eyewear selection system that is easy 
to use and is otherwise "user friendly." 

It is yet still a further object of the present invention 
to provide a structure for an eyeglass frame image and to 
provide methods for creating such a structure. 
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These and other objects are attained in accordance with the 
present invention wherein there is provided an interactive 
eyewear selection system, comprising: means for acquiring a 
digital" image of a person »s face, including the temple areas of 
5 the face; means for storing digital eyeglass frame images, said 
storage means containing a plurality of eyeglass frame images; 
means for selecting an eyeglass frame image from said storage 
means, said image containing frame temple' portions; means for 
superimposing the selected frame image on the image of the 

10 person's face, in a composite image, such that the frame appears 
to be worn naturally on the person's face and the frame temple 
portions are shown at the temple areas of the face; and means 
for displaying the digital image of the person's face and for 
displaying the composite image. 

15 In. accordance with the present invention there is provided 

a method of combining an eyeglass frame image with a face image. 
The method comprises the steps of: (a) obtaining a digital image 
of a person's face; (b) selecting a digital eyeglass frame 
image, the frame image having left and right temple segments; 

2 0 and (c) digitally combining the frame image with the face image 
to produce a composite image that portrays an eyeglass frame 
superimposed on a face, the eyeglass frame portrayed in the 
composite image having left and right temple segments that 
correspond the left and right temple segments of the frame 

2 5 image. 

In accordance with another aspect of the present invention, 
there is provided a method of detecting the irises and pupil 
centers of a person, in a digital image that includes the face 



* » 

wo 00/16683 PCT/US99/2J918 

8 

of the person. The method comprises the steps of: (a) providing 
a gray level image that encompasses the eyes of the person, one 
eye being in a left area of the image and the other eye being in 
a right area of the image; (b) converting the gray level image 
5 to a binary image based on a dynamic threshold T; (c) searching 
the left area of the binary image for black objects and 
calculating the size and centroid coordinates of each black 
object found; (d) searching the right area of the binary image 
for black objects and calculating the size and centroid 

10 coordinates of each black object found; and (e) selecting a 

black object as a candidate for the iris in each of the left and 
right areas of the binary image, in accordance with the 
following selection criteria - (i) the black object selected in 
the left area is approximately the same size as the black object 

15 selected in the right area, and (ii) the black object selected 

in the left area is approximately at the same level as the black 
object selected in the right area. 

In accordance with another aspect of the present invention, 
there is provided a method of detecting the shape of a person's 

20 face in a digital image that includes the face of the person. 

The method comprises the steps of: (a) determining the contour 
of the person's face in the digital image; (b) converting the 
person ' s face contour into the frequency domain by a Fourier 
transformation of a space domain function representing the 

25 contour; (c) providing a plurality of standard face contours 

represented in the frequency domain, each of the standard face 
contours having a name associated therewith; (d) comparing the 
frequency domain representation of the person's face contour 
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with each of the plurality of standard face contours, to 
determine the best match; and (e) assigning the name of the 
standard contour that was determined to be the best match in 
step (d) , to the person's face contour. 
5 In accordance with another aspect of the present invention, 

there is provided a method of determining the frame parameters - 
monocular PDs, ocular centers, and seg-heights, automatically 
from a digital image of a person's face and from a digital image 
of a selected eyeglass frame. 

10 In accordance with another aspect of the present invention, 

there is provided a method of pre-processing digital images of 
eyeglass frames for storage in a frames database, and a 
particular data structure for the frame image. The frame 
processing method includes pre-processing for opaque, semi- 

15 transparent and transparent frames, and for the nosepad portion 
of the frame. Size scaling factors for the rims and bridge of 
the frames are calculated so that only one size need be stored 
in the frames database. Different size frames of a particular 
style are created digitally, based on the calculated scaling 

2 0 factors. 

In accordance with a further aspect of the present 
invention, there is provided a method of selecting eyeglass 
frames in the interactive eyewear system by barcode scanning or 
manually entering an SKU # into the system. The frames database 
25 is indexed by SKU #s. 

In accordance with yet another aspect of the present 
invention, there is provided a method of digitally superimposing 
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color contact lenses on a person's eyes, in a digital color 
image of the person's face. 

BRIEF DESCRIPTION OF THE DRAWING 

Further objects of the present invention will become 
5 apparent from the following description of the preferred 
embodiments with reference to the accompanying drawing, in 
which : 

FIG, 1 is a system block diagram of the interactive eyewear 
selection system of the present invention; 
10 FIG. 2 is a drawing illustrating a digital face image 

containing a number of points determined by the system in FIG. 
1; 

FIG. 3 is a drawing illustrating a digital frame image 
after pre-processing by the system in FIG, 1; 
15 FIG. 4 is a drawing of a composite digital image, showing 

the superimposition of a frame image onto a face image; 

FIG. 5 is a flow diagram outlining the process by which the 
system in FIG. 1 superimposes a frame image onto a face image; 

FIG. 6 is a drawing of a composite digital image containing 
2 0 a number of points and other data for automatic measurement of 
frame parameters by the system shown in FIG. 1; 

FIG. 7 is a flow diagram outlining the process by which the 
system in FIG. 1 automatically determines the frame parameters 
needed to manufacture eyewear; 
25 FIG. 8 is a flow diagram outlining the process by which the 

system in FIG. 1 automatically detects the pupil centers and 
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iris radii of a person's eyes, from a digital image of the 
person's face; 

FIG. 9 is a flow diagram outlining the process by which the 
system in FIG. 1 automatically detects certain points at the 
edge of a person's face, from a digital image of the person's 
face; 

FIG. 10 is a drawing of a digital face image illustrating 
how the system in FIG. 1 automatically detects the contour of a 
person's face from the digital face image; 

FIG. 11 is a flow diagram outlining the process by which 
the system in FIG. 1 automatically detects the shape of a 
person's face, from a digital image of the person's face; 

FIG. 12 is a drawing illustrating a digital frame image 
prior to pre-processing by the system shown in FIG. 1; 

FIG. 13 is a drawing illustrating a digital binary image of 
a regular eyeglass frame; 

FIG. 14 is a drawing illustrating a digital binary image of 
a transparent eyeglass frame; 

FIG. 15 is a drawing illustrating a color flood-filled 
image of an eyeglass frame, in accordance with the present 
invention; 

FIG. 16 is a flow diagram outlining the process by which 
the system in FIG. 1 pre-processes digital eyeglass frame images 
for storage in a frames database; 

FIG. 17 is a flow diagram outlining the process by which 
the system in FIG. 1 pre-processes digital eyeglass frame images 
having. a transparent or partially transparent nosepad; 
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FIG. 18 is a flow diagram outlining the process by which 
the system in FIG. 1 determines a special transparency 
coefficient for representing varying degrees of transparency in 
a frame; 

FIG. 19 is a flow diagram outlining the process by which 
the system in FIG. 1 automatically changes the size of a frame 
in accordance with the manufacturer's size specifications; 

FIG. 2 0 is a flow diagram outlining, the process by which 
the system in FIG. 1 automatically selects a frame image from 
the frames database based on a SKU # scanned in with a barcode 
scanner; 

FIG. 21 is a drawing illustrating a digital image of an eye 
and the parameters used to simulate the superimposition of a 
contact lens on the eye; and 

FIG. 22 is a flow diagram outlining the process by which 
the system in FIG. 1 simulates the superimposition of color 
contact lenses on the eyes, in a digital face image. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring now to FIG. 1, there is shown a block diagram of 
the eyewear selection system 10 of the present invention. 
System 10 is configured in two subsystems. The first subsystem 
is designed to take high resolution, full color digital 
photographs of eyeglass frames, and to create an eyeglass frame 
database to be used in the second subsystem. The second 
subsystem interfaces with the user to acquire a high resolution, 
full color digital photograph of the user's face, and to 
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digitally superimpose a frame image from the database on the 
image of the user's face. 

As shown in FIG. 1, the first subsystem (or "frame image 
processing system") includes a still digital camera 12, a 

5 utility computer program 14 for pre-processing the digital 
photographs of the eyeglass frames, and an eyeglass frame 
database 16. Digital camera 12 should have a picture resolution 
of about 600 X 240 pixels. A suitable still digital camera is 
the DC40, commercially available from Kodak, Rochester, New 

10 York, The eyeglass frame photographs are taken in a 24 -bit 

color format. Program 14 pre-processes the frame photograph, 
derives certain frame data from the pre-processing, and records 
the photograph with the frame data in database 16 . Program 14 
will be described in greater detail hereinbelow. In FIG. 1, 

15 database 16 is conceptual - the database is actually stored in 
the system »s hard drive memory. 

The second subsystem (or "interactive eyewear selection 
system") employs a high resolution CCD color video camera 18 for 
acquiring the image of a person's face. Camera 18 should have a 

20 photo size/resolution of 440 X 440 pixels. A suitable video 

camera for this application is the VCD-510C color video camera, 
by ProVideo . 

An image (or frame) grabber card 2 0 receives the video 
output of camera 18 and converts it to a standard 24 -bit color 
; 25 digital image. A suitable image grabber is the Marvel- 

" Millennium board, by Matrox. The digital image generated by 

grabber 20 is stored in a color image (r,g,b) memory 22, which 
is part of the system's hard drive. At the same time, the 
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digital image is converted back to analog form by a D/A 
converter 24 for display on a touch screen video monitor 26. 

Monitor 26 provides direct interactive capability to the 
user, allowing the user to navigate through the system and make 
5 selections by touching the screen. In this preferred 

embodiment, the touch screen is the primary interface mechanism 
between the user and the system. This feature contributes to 
the "user friendly" aspect of the system. An alternative 
embodiment may employ speech recognition technology to 
10 accomplish interactive functions between the user and the 
system. 

A keyboard 2 8 and barcode scanner 3 0 are also provide as 
input devices to the system. Keyboard 2 8 may be used to input 
commands for controlling system 10 and its frame image 
15 processing and eyewear selection functions. Barcode scanner 30 
is used with a standard program to read, into the system, 
barcoded SKU numbers from actual eyeglass frames. Once entered 
into the system, the SKU number is used to retrieve the 
corresponding frame image from database 16. This process will 
2 0 be further described below. 

A printer is included to print out various forms of data 
and to provide the system operators and users with hardcopy of 
frame images, face images, and composite (frame on face) images. 
A pair of speakers 34 are used to generate audio messages and 
25 instructions to the user. This feature contributes to the "user 
friendly" aspect of the system. 

Both subsystems of system 10 are operated under the control 
of computer processing unit 36. In the preferred embodiment. 
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computer processing unit 3 6 includes a general purpose personal 
computer 38, employing a Pentium microprocessor. Computer 3 8 
contains a number of processing and utility programs for 
carrying out the various jobs and functions of both subsystems 
5 of system 10. The processing methods embodied in these programs 
will be described in detail hereinbelow. 

Systeifl Calibration 

Before system 10 can accurately superimpose a frame image 
onto a face image, and calculate relevant frame parameter data, 
10 system 10 must be initially calibrated. The key here is to 
obtain the actual physical scale for the face image. The 
procedure is as follows: (1) the distance D between video 

camera 18 and the location where the person will sit when the 
camera acquires his or her face image, must be fixed; (2) an 
15 eyeglass frame with known size (i.e., "EYE") is worn by an 

operator who sits at the pre-set image acquiring location, and 
the operator »s facial image is acquired by camera 18 - the 
precise distance D must be maintained; (3) two adjustable 

vertical bars are computer generated on touch screen 26, and are 
2 0 adjusted by the operator so that one is at the left temple and 
the other is at the right temple of the frame (i.e., the front 
of the frame is closely situated between the two vertical bars) , 
and then processing unit 36 calculates the distance d between 

those two bars internally and the frame size parameters, EYE 
2 5 and DEL, are entered from keyboard 2 8 by the operator; and (4) 
the physical scale for system 10 is calculated using the 
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equation, s=d/ {2-EYE+DBL) , and all subsequent physical size 
calculations performed in system 10 will be based on scale s. 

Superimposition of Frame on Face 

Superimposing an eyeglass frame image on a user's face 
image, to create a composite image, is one of the primary 
aspects of the interactive eyewear selection system. The 
objective is to naturally blend the frame image with the user's 
image, while maintaining the original frame image quality. 

To begin with, a user's face image is acquired from video 
camera 18. Still digital camera 12 has taken high quality 
digital photographs of a number of eyeglass frames . As a result 
of the high resolution capability of cameras 12 and 18, the 
actual design patterns on the eyeglass frames will be clearly 
visible in the composite image. This is important to the 
user /customer who is attempting to select a frame for purchase. 

The frame images should be processed with great care. The 
background of the image should be removed smoothly and 
completely, especially the intersection between the background 
and the actual frame depiction. Otherwise, the frame on the 
face, in the composite image, will look fake. The necessary 
frame image pre-processing is performed under the control of 
utility program 14, which will be described in detail below. 
Here we assume that the frame image has been processed properly 
and the user's image is of high quality. 

Referring to Fig. 2, some essential face image parameters, 
such as pupil centers (PI, P2) , interpupil distance (P^) , and 
face edges (Tl, T2) , have been detected (or calculated) and 
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recorded in accordance with "pupil center" and "face edge V 
detection" algorithms, to be described in detail below. As to 



the frame image, some key frame parameters and information have 
been calculated or determined, as shown in Fig. 3. These frame 
5 parameters and information are recorded in database 16 . 

To superimpose the frame image (represented in FIG. 3) on 
the face image (represented in FIG. 2), four critical issues 
must be addressed. First, to make the frame look natural on the 
face, the frame data must be blended into the face properly, 
10 especially at the rim edge. Otherwise, there is an abrupt jump 
from the frame to the face. This jump destroys any realistic 
impression in the composite image. Therefore, a blending 
coefficient a(x,y) is defined. The composite image C is 

generated according to the following equation: 
15 C (x,y) =a (x,y) -face (x,y) + [1-a (x,y) ] -frame (x,y) (1) 

where face(x,y) is the face image and frame (x,y) is the frame 
image , and 



where a (x, y) = ^ 



0; (x,y) e frame 

0.6; {x,y) € frame edge 



As indicated by the above expression, the blending coefficient 
a(x,y) is assigned to the frame points, to the frame edge 
20 points, and to the flood-filled, red, yellow and green, 

background points of the frame image (See FIG. 3 and discussion 
below under the "Frame Image Processing" section) . The value of 
the blending coefficient is set to 0 for the frame points, to 1 
for the background points (r,y,g), and to 0.6 for the frame edge 
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points. The value of a(x,y), for the frame edge points, may be 
set in a range of about 0.5 to about 0.7. 

Before the frame and face images are blended, a 5 x 5 pixel 
neighborhood is checked for each location {x,y) in the frame 
5 image. If a frame point is detected, background points b are 

counted in the neighborhood. If b > 10, the location (x,y) is 

considered the frame edge, and a(x,y) is set to 0 . 6 . If the Jb > 

10 condition is not satisfied, a(x,y) is set to zero. a(x,y) is 
set to 1 for the background points . 

10 The appearance of the frame temples are very important to 

make the composite image look natural . One could imagine what a 
composite image would look like if a frame, without temples, is 
put on a bald head, for example. It is totally unacceptable. 
Therefore, by putting temple information in the composite image, 

15 a significant advancement is achieved by the present invention. 
As a result of face edge detection and pupil center 
detection algorithms (to be described in detail below) , we know 
the positions of face edge points Tl and T2 (See FIG. 2) . The 
frame temples should appear from those two points. We also know 

20 the position of interpupil distance (or pupil midpoint) . 
Also, from the pre-processed frame image (FIG. 3), we know 
temple points Ql and Q2, frame center point f^, and temple 
segments LI and L2 . 

Based on earlier calibration, the frame image is correctly 

25 scaled down and superimposed on the face image. The frame image 
is superimposed on the face image by aligning frame center point 
fc (in the frame image) over pupil midpoint (in the face 
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image) , The frame image may be adjusted relative to the face 
image by an automatic adjustment factor e and/or a manual 
adjustment factor 5. These adjustment factors will be described 
further below with reference to the "Frame Parameters" section. 
5 The left and right temple points Ql and Q2 of the frame 

image are mapped onto the face image, to define left and right 
temple points ql and q2, respectively, on the face image. Then 
left and right temple lengths ql-Tl and T2-q2 are calculated for 
the face image. When the frame image is scaled to the face 

10 image, left and right temple segments LI and L2 are scaled to 
the temple lengths ql-Tl and T2-q2, respectively. The frame 
image is combined with the face image to produce a composite 
image, in accordance with equation (1) above. The scaled temple 
segments of the frame image appear in the composite image 

15 between points ql and Tl and points T2 and q2, respectively, as 
shown in FIG. 4. 

FIG. 5 is a flow diagram illustrating the general process 
of superimposing a frame image on a face image, in accordance 
with the present invention. 

20 To show the temple segments naturally in the composite 

image, the temple segments are faded out gradually from ql, q2 
to Tl, T2, respectively. This is done by assigning a fading 
coefficient ai(x,y) to the scaled temple segments. Thus, the 
composite image at the temple segments is determined by the 

25 following equation: 

composed image^aj (x,y) -face^ll-aj (x,y)] -frame (2) 

where (x, y) -^0 , 6' [x-T^ (x)] / ig^ (x) -T^ (x)] (for left temple) 

aj (x,y) = 0 . 6' [X'T2(x)] / (g2(x) 'T^ (x)] (for right temple) 
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The 0.6 factor in the above expressions was determined 
empirically. This factor may be in the range of about 0.5 to 
about 0.7. 

If the user adjusts (directly on touch screen 26) the frame 
5 position on the face, in the composite image, the temple lengths 
and positions are re -calculated correspondingly. 

Representing the nosepad portion of the frame, and 
transparent frames, in the composite image require special 
attention. The nosepad area seems to be a trivial problem. 
10 However, the nosepad is usually made from transparent material. 
When a frame picture is taken, the nosepad area will pick up the 
background color, even if the background is white. The nosepad 
area will retain the background color in the composite image, 
instead of skin tone. It makes the composite image look fake. 
15 To overcome this problem, during frame pre-processing (see 

details below under the File Nosepad section) , we reserved a 
particular color N =(235,235,235) for the transparent area of 

the nosepad. When we put the frame image on face image, the 
transparent area of the nosepad is converted to the following 
2 0 color - 

nosepad = 0.8' face + 0.2-N (3) 

Here, the transparent area of nosepad is only transferred to 
color i\;= (235, 235 , 235) . The rest of the nosepad color, such as 

shadow and any metal part, are still retained. Otherwise, the 
25 nosepad would look fake as well. The color N was empirically 

determined. A detailed discussion of pre-processing for the 
nosepad is provided below. 
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Transparent frames also need special attention. Otherwise 
the background color will appear in the composite image. Most 
transparent frames have a color pattern where the frame is 
partially opaque or semi-transparent in certain portions. 
Therefore, nosepad approach is not suitable. In this case, a 
blending coefficient c(x,y) is created to represent the 
percentage of transparency at each pixel in the frame image. 
This is part of the pre-processing of the transparent frame ' 
image, which will be described in greater detail below. The 
composite image for a transparent frame is determined by the 
following equation : 

C (X, y) = [c (x, y) /255] 'face (x, y; + -c (x, y) /255] 'frame (x, y) (4) 

■PT-aTng* Paraitieters 

In order to manufacture eyewear, certain frame parameters 
are required. These parameters include the left and right 
monocular pupil PD, the left and right ocular centers, and left 
and right seg-heights . Currently, in optical industry, these 
parameters are measured manually or with the help of some 
devices. With those burdens, it is impossible to obtain high 
accuracy and automation for eyewear manufacturing. The present 
invention provides a unique method for automatically determining 
the above-mentioned parameters. Such automatically derived data 
can be transferred to a laboratory through modem, LAN, WAN or 
internet'. The eyewear can be manufactured directly by using 
such data . 

As mentioned previously, pre-processed frame images (See 
FIG. 3) have been stored in database 16, Several parameters of 
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the frames, such as d^, and f^, have been calculated in 
utility program 14, and stored in database 16. Frame image pre- 
processing is described in greater detail below under the Frame 
Image Processing section. 

Assume that eyewear selection system 10 is calibrated 
during system setup. After the user's face image is acquired by 
the system, the physical scale of the photo is determined. As 
described in the Pupil Detection section below, the pupil 
centers (PI, P2) , interpupil distance Pc/ and iris radii (R-j, R^) 
are automatically detected or calculated. At the same time, the 
frame image is properly scaled to the face image. With good 
calibration, a one -millimeter tolerance can be achieved. 

Once the user selects a desired frame, the system 
superimposes the frame on the user's face image. The frame 
superimposed on the face image has been re-sized according to 
the physical scale. Then the user is able to adjust the 
position of the frame, if necessary, by touching screen 26. For 
example, refer to Figs. 6 and 7. When the frame is superimposed 
on the face, the system aligns the frame center of the frame 
image with the pupil midpoint of the face image. To best 
position the frame on the face, sometimes, a small shift or 
adjustment e is introduced during superimposition . This e is a 
pre-defined constant. In most cases, the frame will be put on 
the face in the correct position. However, if the user wishes 
to manually adjust the frame position on the face, the system, 
can trace such adjustments with the factor 6. Without manual 
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adjustment, 5=0. From the above data, the frame parameters are 

easily calculated in accordance with the following equations: 

Monocular PD: Left = abs(Pc - PI + e (x) + 6 (x) ) 

Right = abs{P2 - P^ + e (x) + 5 (x) ) 

5 Ocular center: O^ = abs(e(y) + 5 (y) + dj) 

Or = abs(e{y) + 5 (y) + dz) 

Seg-height: Si = abs(e(y) + 5 (y) + di - 

= abs(e(y) + 5 (y) + ds - Rr) 

Where x and y stand for horizontal and vertical coordinates, and 
10 where it is assumed that the centroids Ci, C2 of the frame rims 

(See FIG. 6) would be aligned with the pupil centers PI, P2 if 

e(x,y) and 5(x,y) are zero. 

FIG. 7 is a flow diagram illustrating the general process 

of obtaining the necessary frame parameters, in accordance with 
15 the present invent ion . 

Pupil Detection 

Pupil detection is a fundamental step for all advanced i 
features included in system 10, such as automatic frame 
parameter determination. In this description, it is assumed ' 

2 0 that the pupil center is the same as iris center. The pupil 
detection method described here is based on image processing 
technology. The initial input is a digital photograph of the 
user's face. When a photograph is taken by system 10, the front 
of the user's face must appear in the center portion of the 

25 photograph. In addition, the eyes must be aligned with a 

horizontal reference, "eye level" line E, computer generated on 
screen 26 (See FIG. 2) . 

Using video camera 18 and frame grabber 20, a true-color 
digital image (24 bit) is acquired. The following pupil 
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detection processing is performed in processing unit 36 under 
the control of a pupil detection computer program. Initially, 
the 24 bit color digital image of the user's face is converted 
to a gray level image using the NTSC standard. Next, a 
5 centerline must be established in the image as a reference. In 
normal operation, the user is asked to position his/her face in 
the center of the screen. However, this is not always done in 
practice. Therefore, the nosetip N (See FIG. 2) is detected in 
the digital image to establish the centerline. The centerline 
10 is drawn vertically through the y coordinate of the nosetip 

point N.. This technique will ensure the establishment of a more 
accurate centerline . 

In order to find the nosetip, a 10 0x8 0 pixel sub- image is 
selected for processing. The sub- image is extracted from the 
15 original image, 80 pixels below eye level and 50 pixels to left 
and right of the image center. Then, the average brightness 
f(x,y) in the sub- image is calculated in a 5x5 pixel 
neighborhood. It is assumed that the nosetip should be the 
brightest point in the sub- image. Therefore, the nosetip 
20 position (Xo,yo) is determined by the following criteria: 

(x,y) when Max [f (x, y) >180] 
If this search fails, the image center is used as a default 
centerline . 

Next, an area s(x,y) is selected with a width 60 pixels 
25 around the eye level. Then, the gray level image in this 60 

pixel strap is converted to a binary image. Before doing this, 
linear mapping is introduced to increase the image contrast. 
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The mapping equation S is: 

S(x,y) = [s (x,y) -Min(s) ] -255/ [Max(s) -Min(s) ] 
To get the best binary image to represent the irises, a dynamic 
threshold T is determined to accomplish this task. To determine 

T, the average brightness of the sub- image A is investigated. 

At the same time, the histogram of the sub- image H is 

calculated. The following criteria have been empirically 
derived for setting an optimal dynamic threshold: 



2^H(i)^^ / 0.125; A>114 

r=to when — <p; where p= jo. 167; otherwise 

i=0 



The binary image is obtained by - 

f 0; S(x,y)<T 
[ 255; otherwise 

10 After converting to the binary image, black objects are 

exhaustively searched in the binary image, in the left and right 
sides of the centerline, separately. The search is processed 
from top to bottom. The 4 -pixel connection method is applied to 
find all black pixels for each individual object. This is 

15 standard image processing technique. The object size A and 

centroid coordinates are calculated for each object found. 

To save processing time, all objects with a size A smaller 

than 100 are thrown away. Then the best candidate for the iris 
is selected according to the following criteria: 



wo 00/16683 



26 



PCT/US99/21918 



1. The black object selected in the left area is 
approximately the same size as the black object 
selected in the right area, abs (Al-Ar) <50 . 

2. The black object selected in the left area is 

5 approximately at the same level as the black object 

selected in the right area, abs (y^-yR) <20 . 

3. The distance between the black object selected in the 
left area is not too far or too close to the black 
object selected in the right area, 100<abs (xl-Xr) <340 , 

10 4. The black object selected in the left area and the 

black object selected in the right area are the most 
symmetric about the centerline, Min [abs (x^-Xr- 
2*centerline) ] . 

After finding the iris candidates, the centroid coordinates of 
15 those objects indicate the pupil centers and their radii. 

However, the pupil center position may not be accurate 
enough. Thus, a procedure is employed to fine tune the pupil 
center positions. The basic assumption here is that iris is a 
circle -like object and we know the approximate positions of the 
20 pupils. Therefore, the most circle-like edge is searched in a 
20x20 pixel neighborhood of each pupil. The edge point, 
E(xo,yo)/ is regarded acceptable as a possible iris edge, if the 
average intensity inside a test circle, with a 2 pixel radius, 
is less than that outside the test circle, otherwise the point 
25 is ignored. When acceptable, the edge point E(Xo,yo) is set to 
black, according to the following equation: 



255; otherwise 



Where f(x,y) is the gray level image intensity, and (Xi^,yiJ and 
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(^out/Yout) points just inside and outside the test circle, 

defined, by: 

(Xin/Yin) = (xo - 2cos(n). Yo - 2sin(ri)) 

(Xout/Yout) = (Xo + 2cos(n), Yo + 2sin(ri)) 
5 where n is the direction from the circle's center to the point 
(xo.Yo) measured with respect to the x axis. After we got those 
circle-like edge points, the circle patterns with different 
radii were used to find the best match. A variable C is used to 
record the match points. When we move the circle pattern around 
10 the neighborhood, we can find the Max (C) and record the circle 
center at (xo,Yo) and radius R. R will be the radius of the 
iris, and (xo,Yo) will be pupil centers PI and P2 . The pupil 
midpoint or interpupil distance Pc is also calculated from PI and 
P2 . 

15 FIG. 8 is a flow diagram summarizing the general process of 

pupil detection in accordance with the present invention. 

Face Edg e Detection 

Face edge detection is a necessary step to mount the temple 
portions of the frame at the correct positions on the face, in 

20 the composite image. It also provides preliminary parameters 

for a facial shape detection method to be described hereinbelow. 
For face edge detection, a 100 pixel wide sub-image is selected 
around the eye level. This sub- image is converted to a binary 
image using the same procedure described above with respect to 

25 pupil detection. In the binary image, there are usually some 

isolated small black objects which may cause problems with face 
edge detection. To overcome this difficulty, the standard 
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morphologic method erosion (See reference book: Fundamentals of 
Digital Image Processing, by Anil K. Jain) is employed to remove 
those isolated small objects. The erosion window size is 3x3. 

After erosion, the binary image is clear enough to allow 
edge detection to begin. Here, it is assumed that the face edge 
at the temple area will comprise vertical edges. Therefore, in 
a first step, all possible vertical edges are detected. Here, 
the vertical edge detection operator 

1,0 
1,0 

is used to convolve the binary image to obtain another binary 
10 image, fj^, to enhance the vertical edges. 

Usually, some vertical edges in f^, may break, into small 

pieces. In order to find long enough vertical, lines, a 
morphologic dilation operation is applied to the image fi,. The 

dilation window is 3x7. After this operation, most of the 

15 vertical lines are reformed. 

Now the search for face edge points Tl and T2 can start. 
For simplification, the search for the left face edge point Tl 
will be described. The same procedure is applied to right side 
edge point. Starting from the left pupil center PI (See FIG. 2), 

20 a search is made to the left, horizontally, to find a face edge 
point Tl in image fi,. Then from point Ti, the . search continues 

upward and downward in a vertical strap having a 6 pixel width, 
to find any connected edge points which define a vertical edge. 
The accumulation of those connected edge points is counted in C. 
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If C> 20, the search is terminated and the point Tj is used as 

the left face edge point Tl (FIG. 2) . Otherwise, the search 
will continue towards the left side of image f^,. The procedure 

is repeated until a long vertical line is found, i.e., C>20. 

5 The face edge detection process of the present invention is 
outlined in the flow diagram of FIG. 9. 

Facial Shape Detection 

The shape of a person's face is an important factor, from a 
cosmetic standpoint, when selecting eyewear. However, in many 

10 cases, it is difficult for the user to identify his/her facial 
shape, and it is inconvenient as well. System 10 accomplishes 
automatic detection of the user's facial shape, in accordance 
with the present invention. Such a feature assists greatly in 
making the correct shape determination and provides a 

15 significant convenience to the user. This feature contributes 
to the "user friendly" aspect of system 10. 

In the facial shape detection process of the present ' 
invention, a facial contour of the user's face is first 
constructed from the user's face image. Secondly, a facial 

2 0 shape determination is made based on the facial contour 

constructed. The contour is actually classified under one of 
five industry standard facial shapes. The classification step 
is a challenge despite having the standard facial shapes stored 
in memory, as a reference. It is difficult to provide a 

25 standard for judging the best match. 

The process begins by obtaining a full color face image of 
the user. A digital photograph of the user's face is taken by 



wo 00/16683 PCT/US99/21918 

30 

camera 18 and directed through image memory 22 to processing 
unit 36, for image processing. Alternatively, a previously 
acquired and stored photograph may be retrieved from image 
memory 22 . As described with respect to pupil center and face 
5 edge detection, the user's face image is converted to a NTSC 
standard gray level image, and the pupil centers (PI, P2) and 
face edge points (Tl, T2) are obtained (See FIG. 10) . 

Next, the contour of the user's face is determined. To do 
so, some key points on the face are identified. The following 
10 steps are applied to find these points: 

1. Find the nose tip N (FIG. 10} . The procedure for 

detecting the nosetip has been described in the pupil" detection 
section, and will not be repeated here. 

2. Find the mouth position MO (FIG. 10) . To do so, a 

15 function f (y) is created using the average brightness (3 pixels 

wide) along the centerline/nosetip from top to bottom. Then the 
second order derivative f" (y) of f (y) is calculated to determine 

the transition point from light to dark (this point is assumed 
to be mouth position MO) . The mouth position will be: 
2 0 Mq = Max(f"(y)); where y > N (below the nosetip) 

3. Find the bottom of the chin C (FIG. 10) . Point C can 

be determined by the expression: 

C = Ma.x(f"(y)); where y > Mq (below Mouth) 

4. Find the top points of chin (Ml, M2) (FIG. 10). To do 
25 this, a horizontal strap around mouth MO, 20 pixels wide, is 

selected. The linear mapping procedure described in pupil 
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detection section is performed on the image to increase the 
contrast. The gray level image is converted to a binary image 
using the dynamic threshold equation, introduced in the pupil 
detection section. In this case, the threshold setting p is set 
5 to only one value, p = 0,25. Searching proceeds to left and 
right from mouth point MO, until the first black point on each 
side is found. These two points are considered the top chin 
points Ml and M2 (FIG. 10) . 

5. Fitting the boundary of the chin (FIG. 10) . After 
10 obtaining the three key points (C, Ml, M2) , those points are 

fitted with a smooth curve. After studying different fitting 
curves, the parabolic -fitting was determined to be the best in 
most cases. As shown in FIG. 10, the curve (cn) (running along 

the chin boundary between points Ml and M2) , establishes a 
15 contour of the chin. 

6. The contour of the cheek (FIG. 10). Straight lines 
(ckl, ck2) are used to connect Tl and Ml, and T2 and M2, 

respectively. These straight line segments establish the left 
and right cheek contours, respectively. The chin contour and 
20 cheek contours make up the contour of the lower portion of the 
face. In most cases, this approach provides a good 
approximation of the user's face edge in the lower portion of 
the face . 

7. Find forehead point F (FIG. 10) . To find the forehead 

25 point, the iris areas are erased, otherwise some phony results 
will be obtained. So, a 50 X 50 pixel neighborhood of the left 
and right pupil centers is set to the background, value (white) . 
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Then, a search begins upward from eye level along the center line 
for the first black point. That point will be the forehead 
point F. 

8. Find upper portion contour (FIG. 10). After the 
5 forehead point F is obtained, the region from eye level to F is 

searched line by line (i) in both horizontal and vertical 

directions (See FIG. 10) . The first black points we meet toward 
left and right from the centerline and from eye level to the 
forehead are considered as the upper contour points f . The 

10 upper contour points define an upper contour fh of the user's 

face . 

The user's face contour is the composite of the chin 
contour {cn) , the left and right cheek contours {ckl,ck2) , and 

the upper contour {fh) , and is designated as the result, R(x,y) 
15 = cn+ckl'^ck2 -^fh. 

To properly classify the user's face contour, R(x,y) is 

compared to the industry standard face shapes. It has been 
discovered that the comparison should be done in the "frequency" 
domain rather than in the space (x,y) domain. R(x,y) is 

2 0 transformed into the frequency domain by using a 2 -dimensional 
Fourier transformation. In the preferred embodiment, the 
Fourier transformation may be a cosine or sine transformation. 
From the transformation, the "frequency" spectrum of R(x,y) is 

obtained. Then, the corresponding frequency component in the 
25 user's face contour and the industry standard contours are 

compared to determine the best match. To save working memory, 
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the 2D cosine transform was chosen to obtain the frequency 
spectrum of R(x,y) . The equation for the discrete cosine 

transform is as follows: 

F{m,n) =I3 2J^(-i'^) cos (inn? / M) cos { jnn / N) 

i=0 j=0 

To compare with standard face contours, S^, the frequency 
5 spectrum S(m,n) of those standards have been stored in the 

-system's memory. Then the standard deviation Di is calculated 
by : 



M • N i,j=o 



22 [F(m,n) -S.{m,n)]^ ; i = 1,2 



The best matching will be j""^ face contour, where 
Dj = Min(D^) 

10 In most cases, the procedure above will provide accurate 

detection of the face shape. However, sometimes, the oval shape 
is very easy to be confused with other shapes. Therefore, some 
additional criteria are applied to avoid possible confusion. 
Those criteria are : 

15 1. If . abs (Tl (x) -Ml (x) ) <5, abs (T2 (x) -M2 (x) ) <5 and (MO (y) - 
PI (y) ) <2- (PI (y) -F (y) ) , the shape is considered square. 

2: If abs(Tl(x) -Ml(x) )<5, abs (T2 (x) -M2 (x) ) <5 and 0.9(Pl(y)- 
F(y) ) < (MO (y) -Pl(y) ) <1.1 (PKy) -F(y) ) , the shape is 
considered oblong. 

20 3. If Tl '(x) >M0 (x) +10 and T2 (x) <M1 (x) - 10 , the shape is 
considered triangle. 

4. If point Tl, T2, Ml, M2 and C are on the periphery of a 
circle, the shape is considered round. 
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Where x and y stand for the coordinates of each point 
respectively. The numerical factors in the above criteria can 
be varied within the following ranges : 

1. abs (Tl (x) -Ml(x) ) <2-8, abs (T2 (x) -M2 (x) ) <2-8 and (MO (y) - 
5 PI (y) ) < [1.8-2.2] • (PI (y) -F (y) ) . (Square) . 

2. abs (Tl (x) -Ml(x) ) <2-8, abs (T2 (x) -M2 (x) ) <2-8 and [0.8- 
1.2] (PI (y) -F (y))< (MO (y) -PI (y))< [0.9-1.3] (Pl(y)-F(y)) . 
(Oblong) . 

3. Tl (x) >M0 (x) + [7-13] and T2 (x) <M1 (x) - [7-13] . (Triangle). 

10 4. Point Tl, T2, Ml, M2 and C are within 3 pixels inside or 
outside the periphery of the circle. (Round) . 

The face shape detection process of the present invention 

is outlined in the flow diagram of FIG. 11. 

TTT-a-mia Tmage Pre-Processina 

15 This section will describe the frame image pre-processing 

methods of the present invention, which are implemented in 
utility program 14 of system 10. The success of eyewear 
selection system depends on the quality of the frame images and 
the ability to provide a large number of such images to the 

2 0 user. The speed at which such frame images can be acquired and 
pre-processed for database storage is a Icey issue in being able 
to offer such an eyewear selection system. An algorithm to 
automatically process frame images is essential for a feasible 
system. 

2 5 The core part of such an algorithm concerns how to remove 

the background of the frame image and reserve frame information 
needed to reproduce and superimpose the image . The boundary 
part of the frame is the key issue. If the frame boundary 
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cannot be kept smooth, the frame will have a jagged (and thus 
fake) appearance in the composite image. 

The frame pre-processing methods of the present invention 
address opaque, semi-transparent and transparent frames. The 
key is how to detect the frame boundary with high accuracy. 
First, the method of processing metal or non- transparent frames 
will be discussed. Of course, this method can be used for 
transparent frame, but with some reduction in accuracy. 

In this first method, the true color (24 bits) frame image 
(See FIG. 12) is converted to gray- level image by NTSC standard 
This conversion is described under the Pupil Detection section 
of this description. To smooth the boundary of the frame, the 
image f(x,y) is averaged in a 3x3 window. The smoothed image i 
designated, f^(x,y). In order to distinguish the frame 
boundaries and remove the background, two edge detection 
operators and are applied to fs(x,y) in the horizontal and 
vertical directions. The resultant image in both directions 
will be: 

FJx,y) = fJx,y)*E^(x,y) 
where 
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♦stands for convolution. 
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The final boundary image is represented by the equation 
E(x,y) = Max[Fh{x,y), F^(x,y)]. 
For most regular frame images, the background is removed 
completely in the image E(x,y). Just the frame boundary 
information is kept. For further processing, we need to convert 
this gray level image E{x,y) to a binary one. Before this is 
done, linear mapping is employed to increase the contrast of 
E(x,y) (as described in Pupil Detection section) , Then the 
binary image B(x,y) is created by: 



In the binary image B(x,y), in the ideal case, only the frame 

boundary is represented by black. The background is removed and 
filled in with white color. This is true when processing most 
regular frames. FIG. 13 illustrates the binary image B(x,y) of 

a regular frame . 

Now, for transparent frames, the binary image B(x,y) is 

obtained by another approach. Because of the transparency of 
the frame, some frame boundary color is very close to the 
background, in the full color 24 -bit frame image. The edge 
detection operators given above are not powerful enough to 
handle this case. In this approach, the color image, rather 
than the gray- level image, is processed directly. 

To remove the background in the "transparent" approach, we 
have to determine which color indicates the background. Pure 
color paper is used as background when taking the frame digital 





otherwise 
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photographs. In the real world, the background will not be a 
single color, even if pure color paper is used. When using pure 
color paper, the background color in the resulting image will 
fall within a relatively narrow band of colors distributed 
5 around the paper color. 

Another fact is that the background color occupies most of 
the area in the frame color image. Thus, the transparent frame 
approach assumes that the background color is the most 
frequently occurring color in the frame image. However, the 

10 statistics of the image cannot be practically obtained because a 
24 -bit true color image has more than 16 million colors. 
Therefore, a 256 color palette P is generated for the frame 
image. In accordance with palette P, we can convert the 24 -bit 
color image to a 256-color pseudo image. It is possible to 

15 create a histogram for a 256-color image. During this 

conversion, the Microsoft Foundation Classes (MFC) library from 
Microsoft® is used. 

According to the palette P, we can find the nearest index 

(i.e., palette color) in the palette for each pixel (r,g,b). 
20 value and generate a histogram H(i) for the index, in the usual 

fashion. Then, the (r,g,b) value corresponding to the most 
frequent index in the palette is considered as the background 
value b. 

However, if the background is to be removed completely, the 
2 5 surrounding values of the background b must be taken into 

. account. For this purpose, a new quantization value of (r,g,b) 
in the 24 -bit image can be obtained, respectively, according to 
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the following equation: 

where y is the quantization coefficient, which is defined by 

5 After this processing, the neighborhood (or band of) colors will 
be classified to the background b. Meanwhile, the foreground 
color stands out from the background. 

By using the new image data F^^g^j^ and palette P, the frame 

image is converted into a 256 pseudo color image I(x,y). Then 

10 the following threshold is used to create the binary image 
B (x,y) . 

[ 255; H{I{K,y) ) > 800 

B(K,y) = \ 

[ 0; otherwise 

The binary image for a transparent frame is illustrated in FIG. 
14. After the binary image B(x,y) is obtained, for either the 

non- transparent and transparent frames, a new image N(x,y) is 

15 created in which the background is removed and the frame data 
are retained. A flood filling technique is used to fill 
different colors into specifically defined areas in image 
N(x,y). For example, as shown in FIG. 15, the left rim area of 

the frame is filled with red, the right rim area is filled with 
20 green, and the background is filled with yellow. The MFC 

Microsoft® library is employed to accomplish this flood filling 
technique . The unfilled area is reserved for the actual frame 
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data. The pixel values are copied from the original color image 
f(x,y). 

In order to flood fill an area, a starting point for each 
area must be identified. In the typical example shown in FIG. 
15, four starting points are need for the left and right rims 
and upper and bottom background areas, respectively. The 
starting points can be determined by the following rules : 

1. For the left rim: 

(x^fYj) = (image width/4, image height/2) 

2. For the right rim: 

(^TfYr) = (image width'3/4 , image height/2) 

3. For the upper background: 

(K^,yJ = (image width/ 2, d/2) , 

where d is the y- coordinate of the first black point in 
B(x,y) searching downwards along the centerline. 

4. For the bottom background: 

(^bfVb) = (image width/2 , _ image width-lO) 

Upper and bottom background areas are considered separately 
because the temple portions of the frame divide the image 
into upper and lower portions. 

5 . For the double bridge case : 

Another special case is double bridge frames, where an 
additional area exists between two bridges (See FIG. 15) . 
We determine the starting point by: 

(^d^Yd) = (image width/ 2, fd+dj ) /2) 

where dj is the y-coordinate of the first black point in 
B(x,y) searching upwards along the centerline. 

To support eyewear selection system 10, additional data 
from the frame image is necessary. For instance, to superimpose 
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the frame on the face, the center point between the left and 

right rims is required (See "Superimposing Frame on Face" 
above) . To calculate the frame temple lengths in the composite 
image, the starting points (Q1,Q2) of the temples in the frame 
5 image are needed. We will now describe how to obtain this data. 
It is assumed that the frame image has been processed and the 
different areas have been filled with a specified color, as in 
Fig. 15. 

First, the rim centers Cj=fXj,yj; and C2=('x^,y^>) are 

10 calculated. Here, the center of the rim is defined as the 

centroid of the rim area (See FIG. 3) . The following equation 
is applied to obtain the rim centers. 

_ i, j£red . _ i,jeRed 

Area (i, jered) ' ^ Area (i, jered) 



_ je green . _ i, je green 

" Area a, jegreen) ' ^ Area {±, jegreen) 



Then the center point is 

fjx.y) = /"fxj + (yj + y^;/2J 

15 The temple starting points Ql and Q2 are determined by: 

Qj = Min(±) - 15; i e red 

Q2 = Max(i) +15; ± e green 

These frame data are shown in FIG. 3. 

To create different sized frames from one frame image (to 
20 be described below), the bridge points Bl and B2 are necessary. 
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They are calculated by the equations: 

Bj = Max(i) +25/ i ^ red 

B2 = Min(i) - 15; i e green 
For frame measurements, the parameters dl and d2 are 
needed. dl and d2 are the distances between and the bottom of 
the rims, respectively. They are defined by the following 
expressions : 

dj = Max (j ) - fa (y) ; J ^ red 

d2 = Max (j ) - (y) ; j e green 

The temple segments LI and L2 are also determined by the 
following expressions ; 

^ Qi(x) - 0 = (x) 

L2 = Image Width - 02(^) 
Thus, all of the necessary parameters are automatically obtained 
for eyewear selection system 10 (See FIG. 3) . Once this data is 
obtained, it is recorded in database 16. The frame image 
processing methods of the present invention, as discussed above, 
are outlined in a flow diagram in FIG. 16. 

No s epad Pr oc ess ing 

As mentioned above under the "Superimposition of Frame on 
Face" section of this description, the nosepad area of the frame 
requires special attention. Most nosepads are made from both 
transparent and opaque materials. The transparent area of the 
nosepad should be determined precisely and then processed. The 
non- transparent areas should be retained. 
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First, two full color digital photographs are taken, one of 
the frame with a pre-set background, f(x,y), and the other of 

the pre-set background without the frame, b(x,y). Both 
photographs are converted to gray level images, fg(x,y) and 
hg(K,y) , in accordance with the NTSC standard. To save 
processing time and obtain a more accurate result, sub- images 
are extracted and processed. The sub-images, Sf(x,y) and 

Si,(x,y)r are defined in the area: 
w/4: < X < 3-w/4 
h/4 ^ y ^ 3-W/4 

where w is the width and h is the height of the image. In a 
neighborhood of 3x3 pixels, we statistic the difference between 
Sf(x,y) and S^(x,y) by the following criteria: 

c = c + 1; if ahs{S^{x,y) - S^{x,y) )> 20 

where c- is initialized to zero for each 3x3 window. 
According to the variable c, we can modify the original image 

f(x,y) by the following: 

f(x,y) = N = (235,235,235); if fg(x,y) > 190 and c^O 

f(x,y) = f(x,y); otherwise 

Here the color N = (235,235,235), is reserved and optimized to 

indicate the nosepad area. When we mount the frame on the face, 
the color N - representing the transparent area of the nosepad - 

is mixed with the skin tone of the face image, in accordance 
with the equation: nosepad = 0.8-face + 0.2-N. 
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The nosepad processing, as described herein, is outlined in 
the flow diagram of FIG. 17. 

Blending Coefficient for Trans parent Frames 

As mentioned above, transparent frames also require special 



attention. Most transparent frames are not fully transparent. 
There are different degrees or percentages of transparency in 
different areas of a so-called transparent frame. When 



otherwise the composite image will not look real. Here, a 
method is described to determine a special blending coefficient, 
c(x,y), which will properly account for the distribution of 

transparency in a transparent frame image. 

First, two full color digital photographs are taken, one of 
the frame with a pre -set background and the other of the pre -set 
background without a frame. Both photographs are converted to 
gray level images. Before the background image is converted to 
a gray level image, it is smoothed/averaged in a 5x5 pixel 
window. The gray level, smoothed background image is denoted 
b^(:K,y), and the gray level frame image is denoted ('x, yj . The 

transparent blending coefficient c(x,y) can be calculated by the 

following equation: 



superimposing a "transparent" frame image onto a face image, the 
transparency percentage or variation must be taken into account, 
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To store the coefficient c(x,y) into a gray-level image, we have 
to normalize it by: 

c(Xry) = 255 • [c(x,y) - Min(c{x,y) ) ] / [Max{c(x,y) - Min{c{x,y) ) ] 

When we superimpose a transparent frame on a face image, this 
coefficient is called from memory to accurately represent the 
5 transparent percentage at each point in the frame, in the 

composite image. The composite image for a transparent frame- is 
determined by the following equation: 

C (Xr y) = [c (x, y) /255] 'face (x, y)'i- [1-c (x, y) /255] 'frame (x, y) . 

The process of determining the blending coefficient for 
10 transparent frames., as described herein, is outlined in the flow 
diagram of FIG. 18. 

Re- sizing Frame Image 

Normally one frame style has several sizes . If an image 
needs to be taken for each size of every style, database 

15 creation and size would become significant problems. To avoid 
such problems, a method was developed to manipulate one frame 
image to simulate different sizes. 

The entire eyeglass frame does not vary proportionally from 
one size to another. Sometimes, only the rim size changes 

20 according to the size information. The bridge size may not 

change from one size to another. Therefore, one cannot simply 
scale one size up or down and achieve accurate results. The rim 
and bridge areas must be independently variable, so different 
scaling coefficients can be applied to those areas. 
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The rim size (EYE) and bridge size (DEL) are provided by 
the frame manufacturers. For example, if one style has three 
sizes, the middle size is selected for photographing. Then, 
from the frame processing, as described above, the bridge points 
5 Br, B2 and frame size Q1-Q2 are determined (FIG. 3) . Then, the 
scale factors for the bridge and rim are calculated as follows: 
scale^in, = (Q2 ' Q^)/{2'EYE + DEL) 

According to these scale factors, a regular image resize method 
10 can be employed to scale up or down the bridge and rims, 

separately. This processes is outlined in the flow diagram of 
FIG. 19. By employing this method, the time needed to create 
the frames database and the size of the database can be reduced 
by at least half to two-thirds. 
15 Selecting Fra m^ap h y SKU Numbers 

In system 10, barcode scanner 3 0 (See FIG. 1) is used to 
read a barcode label affixed to an actual frame. The barcode 
label contains the SKU number and other unique data for that 
frame. A scanner program decodes the barcode to determine the 

2 0 particular SKU number. Frames database 16 is indexed by the SKU 
numbers of the frames stored therein. Thus, a particular frame 
image can be selected from database 16 by simply scanning in the 
SKU number or manually entering it through keyboard 28. 

An optician's customer who has selected a particular frame 

25 from the optician's display board, can scan the SKU number into 
system 10 to select that particular frame from database 16 . 
Once the customer's face image is acquired by system 10, the 
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customer can view, on screen 26, how he or she looks wearing the 
selected frame. This method is particularly advantageous for 
those who have an extreme case of nearsightedness or 
astigmatism, and require their prescription lenses to view a 
5 frame. In other words, the system can be used as an advanced 
mirror . 

This method of selecting frames from system 10 is outlined 
in the flow diagram of FIG. 20. 

Lens Simulation 

10 System 10 also simulates the optical effects of eyeglass 

lenses installed in the frames. Thus, a customer's prescription 
can be entered into system 10, and the optical effects of this 
prescription can be simulated in the frame that appears in the 
composite image. Thus, the customer can consider the optical 

15 effects of the prescription in a particular frame before 

selecting that frame. Further, opticians can use the lens 
simulation techniques in system 10 to demonstrate to his/her 
customers various lens features and options. 

System 10 includes a comprehensive simulation of lens 

2 0 optics and characteristics found in modern eyewear today. The 
system simulates prescriptions, lens weight and thickness, 
scratch coating and impact resistance, tint color, anti- 
reflection coating, and multi-focal lenses. 

The system takes all standard eye exam prescriptions, such 

25 sphere, cylinder power and prisms. According to such data, the 
system calculates the lens curvatures and corresponding 
distortion on the naked eye area. For nearsighted or farsighted 
patients, a particular distortion parameter X is calculated 
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according to their prescription. The image behind the lens area 
is enlarged or shrunk corresponding to this parameter A. If 
bifocal or trifocal areas exist, different matching parameter Xi 
and As will be calculated. The image behind those areas is 
5 distorted according to those parameters. 

According to standard optics equations and the customer's 
prescription, system 10 can calculate lens curvatures exactly. 
With known material density, the lens weight can be obtained and 
shown after the customer has selected a particular lens 

10 material. By using 3D graphic technology, a 3D view of the lens 
can be drawn, especially the edge thickness. This simulation 
will help the customer select lens material. 

System 10 can simulate scratch resistant coatings. To 
simulate different degrees of scratching, the system mixes a 

15 pre -generated scratch pattern with the lens area, at a 

corresponding percentage. To simulate impact resistance, an 
animation was created. In the animation, a piece of lens is 
clamped by a clamp. While the clamp is screwed tighter and 
tighter, the lens will crack at certain degrees of pressure, 

2 0 corresponding to the lens material. 

The system can simulate lens tint. Regular lenses can be 
tinted with different colors. To show the user the tint effect, 
the system blends some pre-generated color on the lens area. As 
the tint percentage varies, the blending coefficient will change 

25 correspondingly. Therefore, light and dark tints can be 

simulated. The same mechanism is used for a mirror tint. In a 
mirror tint, a pre-generated pattern is used instead of solid 
color. In a gradient color tint, the blending coefficient c(y) 
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is linearly changed according to the following equation: 
c{y) = (y2 - y)/(y2 - Yi) 
whBrG yi = Min(y); y e lens area 

y2 = 2'Max(y)/3; y 6 lens area 
5 The same mechanism is applied to a transition/photochromatic 
lens simulation performed by the system. The transition lens 
changes when indoors and outdoors (partially sunny and full sun) 
are simulated. 

To simulate anti-reflection, a pre-generated reflection 

10 pattern is used to blend with the lens area in reflection cases . 
With an anti-reflection coating, the system simply ignores this 
pattern when creating the lens area image. The system displays 
a comparison, before and after anti-reflection coating. At the 
same time, the night driving benefit of anti -reflection coatings 

15 is simulated by viewing through the lens with and without anti- 
ref lection coating . 

In multi- focal simulations, certain patterns to describe 
multi- focal characteristics are pre-generated, such as executive 
bifocals and flat top bifocals. Since the seg-heights are 

2 0 calculated when determining frame parameters, the multi -focal 
patterns can be placed at the correct position on the lens, 
according to user's choice. At the same time, to show the 
optical effect at the multi-focal area, a slightly different 
blending coefficient c(y) value is used to mix lens and pattern, 

25 so that the multi-focal area is enhanced. 
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Contact Lens Simulation 

System 10 also simulates contact lenses on the eyes. There 
are a few software packages on market today that simulate color 
contact lenses. However, the user must manually indicate or 
5 trace the boundaries of the irises before the contact lenses are 
simulated. This is very inconvenient for the user. System 10, 
on the other hand, automatically places the simulated contact 
lenses on the irises of the eyes in the face image. This can be 
done because system 10 performs automatic pupil center and iris_ 

10 radius detecti^on^pn the face image. ^ 
Because the pupil center and iris radius are^nown, /^he 
system can define the iris area very well with this information. 
Also, the color of the contact lenses is pre-defined. 
Therefore, the key issue is how to blend the iris area with the 

15 pre-defined color for the selected contact lens. The data for 
the color contact lenses are extracted from samples of color 
contacts provided by manufacturers. The data are recorded in- 
database 16 . 

Initially, a digital image of the user's face is acquired 
20 by system 10 in accordance with the present invention, as 

previously described. The pupil centers P1,P2 and iris radii 




with the present invention as previously described. As shown in 
FIG. 21, an outside ellipse 102 and an inside circle 104 are 
25 defined at the eye areas, in the face image. Ellipse 102 

surrounds the iris and establishes the outside boundary for the 
contact lens (i.e., contact lens color). The outside boundary 
is chosen to be an ellipse because most users will squint a 
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little when their picture is taken by video camera 18. The 
squinting reaction causes the iris to elongate slightly and take 
on the shape of an ellipse. Therefore, an elliptical boundary 
provides a better simulation for the iris area than a circle. 
5 In the preferred embodiment, the ellipse for each eye is 

defined as a function of the iris radius R for that eye. In the 
preferred embodiment, the major radius rz of ellipse 102 (See 
FIG. 21) is chosen to be 1.4 times the iris radius R, i.e., T2 = 
1.4-R. The minor radius of ellipse 102 is chosen to be the iris 
10 radius R (See FIG. 21) . In the preferred embodiment, the radius 
ri of inside circle 104 is chosen to be 0.4 times the iris radius 
R, i.e., ri = 0.4-R, As shown in FIG. 21, ri is chosen so that 
circle 104 bounds the pupil of the eye. 

With these parameters defined, a composite image of the 
15 user's face image and the selected contact lens color is 

created. The composite image is created by mixing the contact 
lens color with the user's face image, within the boundaries of 
ellipse 102. In the preferred embodiment, the composite image 
is determined by the following equation: 
20 Composite Image = face ' coef + (1 - coef) - color 

where coef is the mixing coefficient defined, for the preferred 
embodiment, as follows: 



coef - 



0.9; ^ < ^1 

0.2 (r - r^) / (r2 -1) +0.8; ^ r^r2 

1; ^2 < ^ 



where = 0.4R; = 1.4R; and R is the iris radius. 
The blended image produced in accordance with the above will 
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look natural. There is no color jump or any artifacts between 
the natural eye and contact lens (or dye) color. The contact 
lens simulation method of the present invention is outlined in 
the flow diagram of FIG. 22. 
5 While the preferred embodiments of the invention have been 

particularly described in the specification and illustrated in 
the drawings, it should be understood that the invention is not 
so limited. Many modifications, equivalents and adaptations of 
the invention will become apparent to those skilled in the art 
10 without departing from the spirit and scope of the invention, as 
defined in the appended claims. 
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CLAIMS 



What we claim is : 

1 1. An interactive eyewear selection system, comprising: 

2 means for acquiring a digital image of a person's 

3 face, including the temple areas of the face; 

4 means for storing digital eyeglass frame images, said 

5 storage means containing a plurality of eyeglass frame images; 

6 means for selecting an eyeglass frame image from said 

7 storage means, said image containing frame temple portions; 

8 means for superimposing the selected frame image on- 

9 the image of the person *s face, in a composite image, such that 

10 the frame appears to be worn naturally on the person's face and 

11 the frame temple portions are shown at the temple areas of the 

12 face; and 

13 means for displaying the digital image of the person's 

14 face and for displaying the composite image. 

1 2 . A method of combining an eyeglass frame image with a 

2 facial image, comprising the steps of: 

3 (a) obtaining a digital image of a person's face; 

4 (b) locating the pupil centers in said face image and 

5 determining the midpoint Pc between the pupil centers; 

6 (c) locating left and right face edge points Tl and Tr in 

7 said face image, and Tr being located at the edges of the 

8 person's face, adjacent to the left and right temple areas, 

9 respectively, of the person's face; 
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10 (d) selecting an eyeglass frame image, said frame image 

11 having frame points, frame edge points, background points, a 

12 frame center point f^, left and right temple points and Qr, 

13 and left and right temple segments Li^ and Lr,- 

14 (e) assigning a blending coefficient a(x,y) to the frame 

15 points, to the frame edge points, and to the background points 

16 of said frame image, the value of said blending coefficient 

17 being set to zero for the frame points, to one for the 

18 background points, and to a value in the range of about 0 . 5 to 

19 about 0.7 for the frame edge points; 

20 (f) scaling, if necessary, either said frame image or , said 

21 face image so that the scale of said frame image is the same as 

22 that of said face image; ^ ^? 

23 (g) aligning said frame image and said face image in 

24 accordance with the frame center point f^ and the pupil midpoint 

25 Pc; 

2 6 (h) mapping the left and right temple points Ql and Qr of 

27 said frame image on said face image, to define left and right 

28 temple points qL and qR, respectively, on said face image, and to 

29 determine left and right temple lengths, qL-TL and TR-qR, on said 

3 0 face image; and 

31 (i) combining said frame image with said face image to 

32 produce a composite image, in accordance with the equation - 

33 C(x,y) = a{x,y) • face(x,y) + [1 - a{x,y)] • f rame (x, y) , where 

34 face{x,y) is said face image and frame (x,y) is said frame image. 
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1 3. The method of claim 2, wherein the scaling step (f) 

2 includes : 

3 scaling the left and right temple segments and Lr of 

4 said frame image to the temple lengths qL-T^ and T^-qR of said 

5 face image, respectively, the scaled temple segments appearing 

6 in the composite image between points qL and Tl and points Tr and 

7 qR, respectively. 

1 4- The method of claim 3, further comprising the steps 

2 of: 

3 . (j) assigning a fading coefficient ai(x,y) to the scaled 

4 temple segments, the fading coefficient being defined by the 

5 expressions - 

6 ai(x,y) = k • [x - Ti,(x)] / [q^ - Tl(x)] (for left segment), and 

7 ai(x,y) = k • [x - Tr(x)] / [qR - Tr{x)] (for right segment), 

8 where k is a constant with a value in the range of about 0.5 to 

9 about 0.7; and 

10 (k) combining said frame image with said face image along 

11 the scaled temple segments in accordance with the equation - 

12 C(x,y) = ai(x,y) • face(x,y) + [1 - ai(x,y)] • frame(x,y), 

13 whereby the temples gradually fade out in the composite 

14 image from the points q^ and qR to the points Tl and Tr, 

15 respectively - 

1 5. A method of combining an eyeglass frame image with a 

2 facial image, comprising the steps of: 

3 (a) obtaining a digital image of a person's face; 

4 (b) selecting an eyeglass frame image, said frame image 
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5 having frame points, frame edge points, background points; 

6 (c) assigning a blending coefficient a(x,y) to the frame 

7 points, to the frame edge points, and to the background points 

8 of said frame image, the value of said blending coefficient 

9 being set to zero for the frame points, to one for the 

10 background points, and to a value in the range of 0<a(x,y)<l for 

11 the frame edge points; 

12 (d) scaling, if necessary, either said frame image or said 

13 face image so that the scale of said frame image is the same as 

14 that of said face image; and 

15 (e) combining said frame image with said face image to 

16 produce a composite image, substantially in accordance with the 

17 equation - 

18 C(x,y) = a(x,y) • face(x,y) + [1 - a{x,y)] • frame (x,y), . 

19 where face(x,y) is said face image and frame (x,y) is said frame 

20 image. 

1 6. A method of combining an eyeglass frame image with a 

2 facial image, comprising the steps of: 

3 (a) obtaining a digital image of a person • s • face ; 

4 (b) selecting a digital eyeglass frame image, said frame 

5 image having left and right temple segments; and 

6 (c) digitally combining said frame image with said face 

7 image to produce a composite image that portrays an eyeglass 

8 frame superimposed on a face, the eyeglass frame portrayed in 

9 said composite image having left and right temple segments that 

10 correspond the left and right temple segments of said frame 

11 image. 
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1 7. A method of detecting the irises and pupil centers of 

2 a person, in a digital image that includes the face of the 

3 person, said method comprising the steps of: 

4 (a) providing a gray level image g(:x,y) that encompasses 

5 the eyes of the person, one eye being in a left area of image 

6 g(x,y) and the other eye being in a right area of image g(x,y); 

7 (b) converting the image to a binary image h(x,y) based on 

8 a dynamic threshold T determined by the expression, 

EHd) f0.125; A>11A 

T-tr,when — — — <.p; where p= { 

^ 255 0.167; otherwise 

9 where p is less than one, and is equal to a first value when the 

10 average brightness A of image g(x,y) is greater than a threshold 

11 brightness value, and is equal to a second value when A is less 

12 than or equal to the threshold brightness value, and where 

13 binary image b(x,y) is determined from the expression, 

[ 0; if g(x,y)<T 
b{x,y)=\ 

[255; otherivj.se 

14 binary image h(x,y) being defined by the left and right areas of 

15 image g(x,y); 

16 (c) searching the left area of the binary image for black 

17 objects and calculating the size and centroid coordinates of 

18 each black object found; 
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19 (d) searching the right area of the binary image for black 

20 objects and calculating the size and centroid coordinates of 

21 each black object found; and 

22 (e) selecting a black object as a candidate for the iris 

23 in each of the left and right areas of the binary image, in 

24 accordance with the following selection criteria - 

25 (i) the black object selected, in the left, area is 

2 6 approximately the same size as the black object selected in 

27 the right area, and 

28 (ii) the black object selected in the left area is 

29 approximately . at the same level as the black object 

3 0 selected in the right area. 

1 8. The method of claim 7, further comprising the steps 

2 of : 

3 (f) locating, in the gray level image g(K,y), a plurality 

4 of circle edge points within a neighborhood of each iris 

5 candidate selected in step (e) ; 

6 (g) converting the neighborhoods of each iris candidate to 

7 a binary image where the circle edge points are set to black and 

8 the other points in the neighborhood are set to white; 

9 (h) comparing a plurality of circle patterns having known 

10 radii to the circle edge points in each of the neighborhoods, to 

11 • determine the best match between the circle edge points and one 

12 of the plurality of circle patterns; and 

13 (i) recording, for each neighborhood, the known radius of 

14 the circle pattern that best matches the circle edge points and 
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15 recording the location of the pattern's center where the best 

16 match occurred, whereby the recorded radii and center locations 

17 of the circle patterns are considered the iris radii and pupil 

18 centers, respectively, of the person's eyes in the digital image 

19 g(Xfy)' 

1 9. The method of claim 7, wherein the selection criteria 

2 in step (e) further includes - 

3 (iii) the distance between the black object 

4 selected in the left area is not too far or too close to 

5 the black object selected in the right area. 

1 10. The method of claim 9, wherein the left and right 

2 areas of image g(x,y) are divided by a centerline, and wherein 

3 the selection criteria in step (e) further includes - 

4 (iv) the black object selected in the left area and 

5 the black object selected in the right area are the most 

6 symmetric about the centerline . 

1 11. A method of detecting the shape of a person's face in 

2 a digital image that includes the face of the person, said 

3 method comprising the steps of: 

4 (a) determining the contour of the person's face in the 

5 digital image; 

6 (b) converting the person's face contour into the 

7 frequency domain by a Fourier transformation of a space domain 

8 function representing the contour; 
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9 (c) providing a plurality of standard face contours 

10 represented in the frequency domain, each of the standard face 

11 contours having a name associated therewith; 

12 (d) comparing the frequency domain representation of the 

13 person »s face contour with each of the plurality of standard 

14 face contours, to determine the best match; and 

15 (e) assigning the name of the standard contour that was 

16 determined to be the best match in step (d) , to the person's 

17 face contour. 

1 12. The method of claim 11, wherein said step of 

2 determining the person's face contour includes the steps of: 

3 (a) locating a nose point at the tip of the person's nose 

4 in the digital image, the vertical coordinate of the nose point 

5 being situated on a vertical centerline; 

6 (b) locating a mouth point at the person's mouth in the 

7 digital image, the mouth point being substantially aligned with 

8 the nose point along the vertical centerline; 

9 (c) locating a bottom chin point at the bottom of the 

10 person's chin in the digital image, the bottom chin point being 

11 substantially aligned with the mouth point along the vertical 

12 centerline; 

13 (d) locating left and right top chin points at the edge of 

14 the person's face, the left chin point being to the left of the 

15 mouth point and the right chin point being to the right of the 

16 mouth point, the left and right top chin points and the mouth 

17 point being substantially horizontally aligned; 
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18 (e) constructing a contour of the chin in the digital 

19 image, by fitting a curve on the bottom chin point and the left 
2 0 and right top chin points; 

21 (f ) locating left and right face edge points at the edges 

22 of the person's face in the digital image, the left and right 

23 face edge points being adjacent to the left and right temple 

24 areas, respectively, of the person's face; 

25 (g) constructing a contour .of the left cheek in the 

2 6 digital image, by connecting the left temple point and the left 

27 top chin point with a substantially straight line; 

2 8 - • (h) constructing a contour of the right cheek in the 

2 9 digital image, by connecting the right temple point and the 

3 0 right top chin point with a substantially straight line; 

31 (i) locating a forehead point on the person's forehead in 

32 a gray level representation of the digital image, by searching 

33 upward along the vertical centerline until a substantially black 

34 point is detected, the detected point being designated the 
3 5 forehead point; and 

36 (j) locating a plurality of upper contour points around 

3 7 the person's forehead in the gray level representation of the 

3 8 digital image, by searching for substantially black points left 

39 and right from the vertical centerline, line by line until 

40 reaching the forehead point, the detected points being 

41 designated the upper contour points, the upper contour points 

42 defining an upper contour, of the person's face; 

4 3 whereby the person's face contour is the composite of the 
4 4 chin contour, left and right cheek contours, and the upper 

4 5 contour. 
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